Skip to content

One shot all reduce & symm mem sync #245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: joydddd/stack/13
Choose a base branch
from
Draft

One shot all reduce & symm mem sync #245

wants to merge 1 commit into from
+309 −5

Conversation

joydddd
Copy link
Contributor

@joydddd joydddd commented Jul 8, 2025 •
edited
Loading

joydddd added a commit that referenced this pull request Jul 8, 2025
@joydddd
One shot all reduce & symm mem sync
19e6b5c 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from cd18d79 to 19e6b5c Compare July 8, 2025 19:07
This was referenced Jul 8, 2025
Merged
Merged
Merged
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jul 8, 2025
@joydddd joydddd changed the base branch from joydddd/stack/11 to main July 8, 2025 20:23
joydddd added a commit that referenced this pull request Jul 8, 2025
@joydddd
One shot all reduce & symm mem sync
2c79dd9 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 19e6b5c to 2c79dd9 Compare July 8, 2025 20:23
@joydddd joydddd changed the base branch from main to joydddd/stack/11 July 8, 2025 20:24
@joydddd joydddd changed the base branch from joydddd/stack/11 to main July 8, 2025 20:29
joydddd added a commit that referenced this pull request Jul 8, 2025
@joydddd
One shot all reduce & symm mem sync
374dfa3 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 2c79dd9 to 374dfa3 Compare July 8, 2025 20:29
@joydddd joydddd changed the base branch from main to joydddd/stack/11 July 8, 2025 20:29
@joydddd joydddd changed the base branch from joydddd/stack/11 to main July 9, 2025 18:19
joydddd added a commit that referenced this pull request Jul 9, 2025
@joydddd
One shot all reduce & symm mem sync
1887088 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 374dfa3 to 1887088 Compare July 9, 2025 18:20
@joydddd joydddd changed the base branch from main to joydddd/stack/11 July 9, 2025 18:20
@joydddd joydddd changed the base branch from joydddd/stack/11 to main July 9, 2025 19:51
joydddd added a commit that referenced this pull request Jul 9, 2025
@joydddd
One shot all reduce & symm mem sync
f3199f8 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 1887088 to f3199f8 Compare July 9, 2025 19:51
@joydddd joydddd changed the base branch from main to joydddd/stack/11 July 9, 2025 19:51
@joydddd joydddd force-pushed the joydddd/stack/11 branch from 6815f03 to 3f492ce Compare July 9, 2025 19:52
joydddd added a commit that referenced this pull request Jul 9, 2025
@joydddd
One shot all reduce & symm mem sync
027a16b 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from f3199f8 to 027a16b Compare July 9, 2025 19:52
@joydddd joydddd changed the base branch from joydddd/stack/11 to main July 9, 2025 21:27
joydddd added a commit that referenced this pull request Jul 9, 2025
@joydddd
One shot all reduce & symm mem sync
2f52133 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 027a16b to 2f52133 Compare July 9, 2025 21:28
@joydddd joydddd changed the base branch from main to joydddd/stack/11 July 9, 2025 21:28
@joydddd joydddd changed the base branch from joydddd/stack/11 to main July 9, 2025 21:30
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 88dbf9c to 7af5d2b Compare July 11, 2025 02:19
@joydddd joydddd changed the base branch from main to joydddd/stack/13 July 11, 2025 02:19
@joydddd joydddd changed the base branch from joydddd/stack/13 to main July 14, 2025 18:18
joydddd added a commit that referenced this pull request Jul 14, 2025
@joydddd
One shot all reduce & symm mem sync
7a3d259 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 7af5d2b to 7a3d259 Compare July 14, 2025 18:35
@joydddd joydddd changed the base branch from main to joydddd/stack/13 July 14, 2025 18:35
@joydddd joydddd changed the base branch from joydddd/stack/13 to main July 15, 2025 18:40
joydddd added a commit that referenced this pull request Jul 15, 2025
@joydddd
One shot all reduce & symm mem sync
da1c594 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 7a3d259 to da1c594 Compare July 15, 2025 18:40
@joydddd joydddd changed the base branch from main to joydddd/stack/13 July 15, 2025 18:40
@joydddd joydddd changed the base branch from joydddd/stack/13 to main July 15, 2025 20:13
joydddd added a commit that referenced this pull request Jul 15, 2025
@joydddd
One shot all reduce & symm mem sync
f78a376 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from da1c594 to f78a376 Compare July 15, 2025 20:13
joydddd added a commit that referenced this pull request Jul 15, 2025
@joydddd
One shot all reduce & symm mem sync
20c357c 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from f78a376 to 20c357c Compare July 15, 2025 20:14
@joydddd joydddd mentioned this pull request Jul 15, 2025
Merged
@joydddd joydddd changed the base branch from main to joydddd/stack/16 July 15, 2025 20:14
@joydddd joydddd changed the base branch from joydddd/stack/16 to main July 15, 2025 22:05
joydddd added a commit that referenced this pull request Jul 15, 2025
@joydddd
One shot all reduce & symm mem sync
cacfce0 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from 20c357c to cacfce0 Compare July 15, 2025 22:05
@joydddd joydddd changed the base branch from main to joydddd/stack/16 July 15, 2025 22:05
@joydddd
Copy link
Contributor Author

joydddd commented Jul 16, 2025 •
edited
Loading

One shot All Reduce Performance on 8xH100.

Benchmark: https://github.com/joydddd/kraken/tree/helion_bench

Performance drop at 1m due to hardcoded block_size.

shape dtype nccl helion_1shot triton_1shot dist_1shot Speedup over nccl Best Backend
(4k) torch.bfloat16 21.408 14.304 13.440 16.032 1.593 triton_1shot
(8k) torch.bfloat16 22.112 14.432 13.280 15.968 1.665 triton_1shot
(16k) torch.bfloat16 23.904 14.912 13.696 16.960 1.745 triton_1shot
(32k) torch.bfloat16 24.032 15.232 14.688 18.080 1.636 triton_1shot
(64k) torch.bfloat16 24.128 17.312 17.088 19.840 1.412 triton_1shot
(128k) torch.bfloat16 24.416 20.704 21.376 25.088 1.179 helion_1shot
(256k) torch.bfloat16 24.736 29.184 29.312 30.400 1.000 nccl
(512k) torch.bfloat16 34.016 47.488 42.016 43.328 1.000 nccl
(1m) torch.bfloat16 55.808 91.168 63.488 65.376 1.000 nccl

@joydddd joydddd changed the base branch from joydddd/stack/16 to main July 16, 2025 03:11
joydddd added a commit that referenced this pull request Jul 16, 2025
@joydddd
One shot all reduce & symm mem sync
a357b61 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd force-pushed the joydddd/stack/12 branch from cacfce0 to a357b61 Compare July 16, 2025 03:11
@joydddd joydddd changed the base branch from main to joydddd/stack/13 July 16, 2025 03:11
@joydddd
One shot all reduce & symm mem sync
eaf93f8 
stack-info: PR: #245, branch: joydddd/stack/12
@joydddd joydddd changed the base branch from joydddd/stack/13 to main July 16, 2025 03:13
@joydddd joydddd force-pushed the joydddd/stack/12 branch from a357b61 to eaf93f8 Compare July 16, 2025 03:13
@joydddd joydddd changed the base branch from main to joydddd/stack/13 July 16, 2025 03:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@joydddd @facebook-github-bot